A method for reconstructing 3D scenes based on the use of convolutional neural networks, filtering by distance and using an “octree” 

Yuriy V. Dubenko, Candidate of engineering sciences, associate professor, associate professor of the sub-department of informatics and computer engineering, Kuban State Technological University (2 Moskovskaya street, Krasnodar, Russia), E-mail: scorpioncool1@yandex.ru
Evgeniy E. Dyshkant, Candidate of engineering sciences, senior lecturer of the sub-department of in-plant electrical equipment and automation, Kuban State Technological University (2 Moskovskaya street, Krasnodar, Russia), E-mail: ed0802@yandex.ru
Nikolay N. Timchenko, Applicant, Kuban State Technological University (2 Moskovskaya street, Krasnodar, Russia), E-mail: north_11@mail.ru
Nikita A. Rudeshko, Postgraduate student, Kuban State Technological University (2 Moskovskaya street, Krasnodar, Russia), E-mail: nikita.rudeshko@yandex.ru 

004.932.72'1 

10.21685/2072-3059-2021-4-4 

Background. The object of research is machine vision systems. The subject of research is the methods of reconstruction of three-dimensional scenes. The aim of the work is to develop a method for reconstruction of three-dimensional scenes, including the stages of identification of three-dimensional objects, segmentation (selection) and filtering of their constituent points in the original cloud. Materials and methods. The Block-Matching Algorithm method used to generate a depth map, the Mask R-CNN convolutional neural network – to identify and segment objects in the external environment, the OctTree method – to filter the point cloud, the Delaunay triangulation method - to generate a threedimensional model. Results. Based on the proposed method for reconstructing threedimensional scenes, a software product was developed in the python programming languages (TensorFlow virtual environment for implementing the Mask RCNN convolutional network) and C #, which implements the formation of a three-dimensional model of the road surface. Conclusions. The resulting three-dimensional model of the road surface can then be used to determine the following parameters: boundaries, axis of the road surface, geometric dimensions of defects (potholes, waves, depressions, chipping, sweating, protrusions, cracks), longitudinal evenness index (IRI). 

three-dimensional scene, reconstruction, segmentation, filtering, recognition, convolutional neural network 

1. Nguen Tkhu Khyong, Nguen Tkhe Long. Algorithmic and software for automatic detection and classification of pavement defects using the graph cutting method and the random forest algorithm. Vestnik Irkutskogo gosudarstvennogo tekhnicheskogo universiteta = Bulletin of Irkutsk State Technical University. 2016;(10):111–118. (In Russ.)
2. Sobol' B.V., Solov'ev A.N., Vasil'ev P.V., Podkolzina L.A. Deep convolutional neural network model in the problem of crack segmentation on asphalt images. Vestnik Donskogo gosudarstvennogo tekhnicheskogo universiteta = Bulletin of Don State Technical University. 2019;(1):63–73. (In Russ.)
3. Maysuradze Sh.G., Maysuradze I.G. Solution of engineering road problems based on mobile laser scanning data. SAPR i GIS avtomobil'nykh dorog = CAD and GIS of highways. 2016;(2):50–53. (In Russ.)
4. Sistema videodefektatsii pokrytiy avtomobil'nykh dorog (Rosdortekh) = Video detection system for road surfaces. (In Russ.). Available at: https://www.rosdorteh.ru/catalog/27/
5. Kaz'min V.N., Noskov V.P. Isolation of geometric and semantic objects in ranging images for robot navigation and environment reconstruction. Izvestiya YuFU. Tekhnicheskie nauki = Proceedings of the Southern Federal University. Engineering sciences. 2015;(10):71–83. (In Russ.)
6. Goritov A.N., Yakovchenko S.I. Building a model of the robot’s environment based on stereo vision. Dinamika sistem, mekhanizmov i mashin = Dynamics of systems, mechanisms and machines. 2017;(1):121–125. (In Russ.)
7. Chugunov R.A., Kul'nevich A.D., Aksenov S.V. A technique for constructing stereo image depth maps using a capsular neural network. Doklady Tomskogo gosudarstvennogo universiteta sistem upravleniya i radioelektroniki = Reports of Tomsk State University of Control Systems and Radioelectronics. 2019;(1):83–86. (In Russ.)
8. Mohan D., Ram Dr. A.R. A Review on Depth Estimation for Computer Vision Applications. International Journal of Engineering and Innovative Technology (IJEIT), 2015;4(11). Available at: https://pdfs.semanticscholar.org/f75b/2030713b36addde9b9d 551ee003c20bac3ca.pdf
9. Barjayta A. Block matching algorithms for motion estimation. IEEE Transactions Evolution Computation. 2004;8(3):225–239.
10. Medvedev M.V., Kirpichnikov A.P. 3D reconstruction of objects in the technical vision system of a mobile robot. Vestnik Kazanskogo tekhnologicheskogo universiteta = Bulletin of Kazan Technological University. 2014;15(15):326–330. (In Russ.)
11. Kaiming He, Gkioxari G., Dollár P., Girshick R. Mask R-CNN. Computer Vision and Pattern Recognition (cs.CV). arXiv:1703.06870 – Submitted on 20 Mar 2017 (v1), last revised 24 Jan 2018 (this version, v3)
12. Liu J. [et al.]. RocNet: Recursive Octree Network for Efficient 3D Deep Representation. arXiv:2008.03875 [cs.CV].
13. Skvortsov A.V. Triangulyatsiya Delone i ee primenenie = Delaunay triangulation and its application. Tomsk: Izd-vo Tom. un-ta, 2002:128. (In Russ.)
14. Patent RU 2650857 C1. System for determining the geometric parameters of 3D objects. Dubenko Yu.V., Timchenko N.N. No. 2017111746; appl. 06.04.2017; publ. 17.04.2018. (In Russ.)
15. Dubenko Yu.V., Dyshkant E.E., Rudeshko N.A., Timchenko N.N. Nauchno-tekhnicheskiy otchet po rezul'tatam vypolneniya etapa po dogovoru № 3617GS1/60550 (kod 0060550) ot 20.07.2020 «Razrabotka tsifrovoy platformy dlya obespecheniya bezopasnosti dorozhnoy infrastruktury s primeneniem trekhmernogo lazernogo skanirovaniya», registratsionnyy nomer IKRBS: AAAA-B20-220122990242-9, zareg. 29.12.2020, 64 s = Scientific and technical report on the results of the stage under the contract No. 3617GS1/60550 (code 0060550) from July 20, 2020 “Development of a digital platform for ensuring the safety of road infrastructure using 3D laser scanning”, registration number IKRBS: AAAA-B20-220122990242-9, registrated on December 29, 2020, 64 pages. (In Russ.)